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VISION SYSTEM FOR A VEHICLE 


CROSS REFERENCE TO RELATED 
APPLICATIONS 


This patent application is a continuation of U.S. patent 
application Ser. No. 11/562,824, entitled “Vision System for 
an Autonomous Vehicle,’ filed Nov. 22, 2006, which claims 
the benefit under 35 U.S.C. §119(e) of U.S. Prov. Pat. App. 
No. 60/805, 197, entitled “Vision System for an Autonomous 
Vehicle,” filed on Jun. 19, 2006, both of which are hereby 
incorporated by reference herein in their entireties. 


BACKGROUND 


Autonomous and semi-autonomous vehicles have the 
potential to be used in an increasing number of applications. 
For example, the use of autonomous vehicles may be desir- 
able in military or civilian applications that would otherwise 
expose human operators or passengers to dangerous environ- 
ments. Such autonomous vehicles may include a control sys- 
tem configured to receive information regarding, for 
example, the surrounding terrain, upcoming obstacles, a par- 
ticular path, etc., and to automatically respond to this infor- 
mation in place of a human operator by commanding a series 
of maneuvers so that the vehicle is able to negotiate the 
terrain, avoid the obstacles, or track a particular path with 
little or no human intervention. 

In certain applications, the autonomous vehicle may be 
required to operate in a continuously changing environment. 
For example, the autonomous vehicle may be required to 
track a path over alternate stretches of flat, winding, or hilly 
terrain at varying speeds, and across which many obstacles, 
such as other vehicles or various other objects, or even 
humans may appear at unexpected places or times. Without 
the presence of a human operator to assess the changing 
conditions, the autonomous vehicle may collide with an 
obstacle or otherwise attempt maneuvers beyond its reactive 
capabilities. In certain other applications, the autonomous 
vehicle may be required to operate without detection, such as 
in a military operation or even at night. Accordingly, it would 
be desirable to provide an improved autonomous vehicle that 
is configured to address these and other concerns. 


SUMMARY 


Autonomous and/or semi-autonomous vehicles are 
described herein. The vehicles may be any suitable type or 
make of vehicle. The vehicle may be a military vehicle or a 
civilian vehicle. In either context, the vehicle may be config- 
ured to transport large payloads over long distances and/or 
off-road. The vehicle may include various subsystems such as 
a throttle system, transmission system, braking system, and/ 
or steering system. In some embodiments, the vehicle may 
include an all-wheel steering system, an independent suspen- 
sion, and/or a compressed air system. The compressed air 
system may be used to operate the brakes and/or other vehicle 
subsystems. 

The vehicle may include a vehicle control system that is 
used to operate the vehicle autonomously. The vehicle control 
system may include a variety of sensors which gather infor- 
mation regarding the status of the vehicle, the environment 
around the vehicle (especially in front of the vehicle), the 
position ofthe vehicle, and so forth. A variety of systems may 
be used to provide information to the vehicle control system 
to allow for autonomous operation of the vehicle. For 
example, such systems may include LIDAR systems (also 
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2 


referred to as LADAR systems in a military context), radar 
systems, vision systems, GPS systems, and so forth. 

The entire control system or various components thereof 
may be provided as part of a kit which is used to retrofit 
existing vehicles to make them autonomous. This may be 
particularly desirable in situations where there are a fleet of 
vehicles (e.g., in a military or civilian context) that have 
similar configurations. For some vehicles that already include 
anelectronic control system, the kit may simply include some 
sensors and additional software (or hardware if needed). For 
those vehicles that are largely controlled mechanically (me- 
chanical steering, braking, throttle, etc.) the kit may include 
additional motors and/or other sub control systems for the 
vehicle. 

The vehicle control system may include a vision system 
that is used to detect obstacles relative to the vehicle and/or 
identify a path for the vehicle to follow. The vision system 
may include a plurality of cameras that are configured to 
receive input in the form of images of the surrounding envi- 
ronment. The vision system may include a processor that is 
configured to receive and process the images to extract useful 
information about the surrounding environment. In a military 
context, one advantage of the vision system relative to other 
obstacle detection techniques such as LIDAR or radar is that 
the vision system does not transmit electromagnetic radiation 
that can be detected and/or jammed by the enemy. 

In one embodiment, the vision system may include at least 
three cameras. The vision system may be configured to 
receive input from various combinations ofthe three cameras 
depending on the status of the vehicle (e.g., speed, steering 
angle, and so forth) or depending on the surrounding envi- 
ronment (e.g., the rate that obstacles are detected, the size of 
the obstacles, and so forth). For example, the three cameras 
may be mounted on the vehicle so that the cameras are asym- 
metrically spaced apart from each other. Thus, the distance or 
baseline between each combination of two cameras is differ- 
ent. The vision system may be configured to use a combina- 
tion of the cameras having a wide baseline in order to detect 
obstacles that are relatively far away. The vision system may 
be configured to use a combination of cameras having a 
shorter baseline to detect obstacles that are closer to the 
cameras. The reason for doing this is that objects that are far 
away from each other can be identified and matched in images 
taken from cameras having a wide baseline. However, if the 
same objects are positioned much closer to the cameras, the 
wide baseline of the cameras results in the objects looking 
very different in the images. Thus, when the objects are 
nearby, cameras having a shorter baseline may be used to 
make it easier to identify and match the object in the images 
taken from the cameras. 

Other features and advantages of the subject matter 
described herein will become apparent from the following 
detailed description and accompanying drawings. It should 
be understood, however, that the detailed description and 
specific examples are given by way of illustration and not 
limitation. Many modifications and changes within the scope 
ofthe present invention may be made without departing from 
the spirit thereof, and the invention includes all such modifi- 
cations. 


DRAWINGS 


The exemplary embodiments will hereafter be described 
with reference to the accompanying drawings, wherein like 
numerals depict like elements, and: 

FIG. 1 is a perspective view of one embodiment ofa vehicle 
that is autonomous and/or semi-autonomous. 
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FIG. 2 is a block diagram schematically illustrating a 
vehicle control system according to an exemplary embodi- 
ment. 

FIG. 3 is a front view of another embodiment of an autono- 
mous and/or semi-autonomous vehicle. 

FIG. 4 is a front view of the vehicle in FIG. 3 which shows 
some of the sensors coupled to the roll cage and/or on or near 
the cab of the vehicle. 

FIG. 5 is a front view of the vehicle in FIG. 3 which shows 
some additional sensors positioned near the bumper of the 
vehicle. 

FIG. 6 is a perspective view of another embodiment of an 
autonomous and/or semi-autonomous vehicle showing sen- 
sors positioned near the roll cage and the bumper. 

FIG. 7 is a perspective view of the vehicle in FIG. 6 show- 
ing two LIDAR sensors coupled to the bumper of the vehicle. 

FIG. 8 is a perspective view of the vehicle in FIG. 6 show- 
ing a LIDAR sensor and a camera coupled to the roll cage of 
the vehicle. 

FIG. 9 is a perspective view of the interior of another 
embodiment of an autonomous and/or semi-autonomous 
vehicle showing at least a portion of the hardware for the 
vehicle control system positioned under the passenger seat. 

FIG. 10 is a perspective view of another embodiment of an 
autonomous and/or semi-autonomous vehicle. 

FIG. 11 is a perspective view of the vehicle in FIG. 10 
showing sensors positioned near the lower front section of the 
vehicle. 

FIG. 12 is a perspective view of the vehicle in FIG. 10 
showing sensors coupled to the cab of the vehicle. 

FIGS. 13-15 are illustrations of the path planning, obstacle 
detection, and obstacle avoidance capabilities of another 
embodiment of an autonomous and/or semi-autonomous 
vehicle. 

FIGS. 16-19 are pictures showing the obstacle detection 
capability of the vision system. 

FIGS. 20A-20C are a series of pictures that show the path 
detection capability of the vision system. 

FIGS. 21-23 are a series of pictures that show the ability of 
the vision system to detect obstacles in adverse lighting con- 
ditions. 


DETAILED DESCRIPTION 


Before turning to the FIGS. which illustrate the exemplary 
embodiments in detail, it should be understood that the sub- 
ject matter described herein is not limited to the details or 
methodology set forth in the following description or illus- 
trated in the FIGS. The subject matter described herein is 
capable of being provided in other embodiments or being 
practiced or carried out in various ways. It should also be 
understood that the phraseology and terminology employed 
herein is for the purpose of description only and should not be 
regarded as limiting. Further, while the various exemplary 
embodiments are primarily described in the context of 
autonomous vehicles, it should be understood that other types 
of vehicles are contemplated as well, such as semi-autono- 
mous vehicles and the like. The term “autonomous vehicle” 
as used herein generally refers to a vehicle configured for 
unmanned operation, i.e., operation without a human pilot or 
co-pilot, with some level of autonomy built in, but which may 
or may not also carry one or more human passengers. Semi- 
autonomous vehicles are those vehicles that operate to some 
degree in an autonomous fashion. For example, a semi-au- 
tonomous vehicle may be one which is capable of receiving 
remote instructions (e.g., via a wireless connection) from a 
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person to operate the vehicle in certain situations such as 
when the vehicle can no longer operate autonomously for 
some reason. 

It should be appreciated that the vehicles, control systems, 
features, advantages, and any other subject matter described 
in: (1) U.S. Pat. No. 6,922,615, entitled “Turret Envelope 
Control System and Method for a Fire Fighting Vehicle,” 
issued on Jul. 26, 2005, (2) U.S. Pat. No. 6,885,920, entitled 
“Control System and Method for Electric Vehicle,” issued on 
Apr. 26, 2005, (3) U.S. Pat. No. 6,882,917, entitled “Steering 
Control System and Method,” issued on Apr. 19, 2005, (4) 
US. Pat. No. 6,421,593, entitled “Military Vehicle having 
Cooperative Control Network with Distributed I/O Interfac- 
ing,” issued on Jul. 16, 2002, and (5) U.S. Prov. Pat. App. No. 
60/723 ,363, entitled “Vehicle Control System and Method,” 
filed on Oct. 4, 2005, may be used and/or incorporated with 
the subject matter described herein. All possible configura- 
tions of the subject matter described in these documents and 
the subject matter described herein is contemplated and 
hereby expressly disclosed. For example, any of the vehicles 
or any portion of any of the vehicles (e.g., vehicle control 
systems, components for vehicle control systems, etc.) in 
these documents with their various configurations, control 
systems, features, and components may be combined with all 
or any part of the subject matter described herein. The docu- 
ments (1)-(5) listed in this paragraph are hereby incorporated 
by reference herein in their entireties as if the entire contents 
of these documents were reproduced herein. 

Referring to FIG. 1, one embodiment of an autonomous 
vehicle 102 is shown. The vehicle 102 is a tactical wheeled 
vehicle that may be used in a military context to transport 
troops, supplies, or any other goods. The vehicle 102 has 
eighteen inches of ground clearance and is capable of travers- 
ing rugged terrain. The vehicle 102 may have features such as 
all-wheel drive, all-wheel steering, independent suspension 
(all or some of the wheels), and a central tire inflation system. 
Many of these features make it easier for the vehicle 102 to 
operate autonomously. For example, since the vehicle 102 is 
relatively large, the addition of all-wheel steering allows the 
vehicle 102 to maneuver quickly and with more agility to 
avoid obstacles, pass through tunnels, etc. Also, the central 
tire inflation system may be used to help the vehicle 102 get 
unstuck by reducing the air pressure in the tires and thereby 
increasing the fraction of the vehicle 102. 

The vehicle 102 may also be configured to handle about 
60% grades and about 30% side slopes while the vehicle 102 
is fully loaded. The vehicle 102 may be capable of carrying 
about a seven ton payload off-road and about a fifteen ton 
payload on-road. The vehicle 102 may also be capable of 
fording five feet of water, traveling up to and above about 65 
miles per hour, and cruising for a range of about 300 miles or 
more. 

The vehicle 102 may be provided in a dump truck configu- 
ration where the bed pivots upward to unload material in the 
bed of the truck. The vehicle 102 may also be configured as a 
wrecker vehicle that is capable of towing disabled vehicles 
such as tanks or other trucks to be repaired. The vehicle 102 
may also include a flat bed and an articulated lifting system. 
The lifting system may be configured to move supplies or 
other goods onto the flat bed. The lifting system folds up near 
the end of the bed in a stowed position while the vehicle 102 
is traveling. The vehicle 102 may include a load handling 
system that is capable of receiving and securely transporting 
standard or custom sized shipping containers. In one embodi- 
ment, the vehicle 102 may be the medium tactical vehicle 
replacement (MTVR) manufactured by Oshkosh Corpora- 
tion, located in Oshkosh, Wis. The vehicle 102 may include 
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any of the various commercial embodiments of the MTVR 
such as the cargo truck embodiment (regular and extended), 
dump truck embodiment, wrecker embodiment, HIMARS 
resupply vehicle embodiment, tractor embodiment, and/or 
load handling system embodiment. 

In a military context, there are some challenges to and 
advantageous of using an autonomous vehicle. For example, 
one of the challenges is that many ofthe technologies used to 
sense the terrain around the vehicle rely on the transmission 
of electromagnetic radiation (e.g., LIDAR, radar). The radia- 
tioncan be detected by the enemy and used as a signal to guide 
bombs or other munitions to the autonomous vehicle. Also, 
the enemy may be able to jam the signals and cause the 
vehicle to crash or otherwise be disabled. The principle 
advantage of an autonomous vehicle in a military context is 
that fewer people need to be put in danger in order to transport 
supplies and goods. 

Although the vehicle 102 is described in the context of a 
military vehicle, it should be appreciated that the subject 
matter described herein may be used with any of a number of 
suitable vehicles regardless of whether they are civilian or 
military vehicles, heavy duty or light duty vehicles, or large or 
small vehicles. For example, the subject matter described 
herein may be used with the firefighting vehicles described in 
some of the documents that are incorporated by reference 
above. 

The vehicle 102 may be configured to be easily switched 
from manual operation to autonomous operation. The 
switchover may be as simple as pressing a button or may 
involve some amount of mechanical or electrical reconfigu- 
ration. Allowing the vehicle 102 to be used in both manual 
and autonomous mode increases the usability of the vehicle 
102. For example, in a military context, if the vehicle 102 is 
inoperable in autonomous mode due to enemy fire, the 
vehicle 102 may still be capable of being operated manually. 
Also, the vehicle 102 may be capable of being operated by an 
operator until the vehicle 102 enters a dangerous area (e.g.， 
vehicle approaches the enemy, vehicle enters a hazardous 
waste area) at which point the vehicle may be operated 
autonomously. 

FIG. 2 is a block diagram schematically illustrating a 
vehicle control system 100 for use with the vehicle 102 
according to an exemplary embodiment. The vehicle control 
system 100 may include a terrain detection system 104, a 
vehicle management system 106, and a system manager 108. 
The vehicle control system 100 also includes a plurality of 
vehicle subsystems 110, position sensors 150, and terrain 
sensors 160. The vehicle control system 100 is generally 
configured to acquire and process vehicle position informa- 
tionand terrain information from the position sensors 150 and 
the terrain sensors 160 to control the vehicle 102 autono- 
mously. 

Referring to FIG. 9, certain hardware components of the 
vehicle control system 100 may be positioned under the pas- 
senger seat 101 of the vehicle 102. The hardware components 
may be packaged in a shock absorbing rack that fits inside the 
passenger seat 101. In one embodiment, the vehicle 102 may 
include a temperature control system that is configured to 
heat and/or cool the hardware components. For example, the 
vehicle 102 may be configured to air condition the space 
under the passenger seat 101 to prevent the microprocessors 
from overheating. The vehicle 102 may be configured to heat 
and/or cool the space that contains the hardware components 
using the heating and cooling systems used to control the 
climate in the cab of the vehicle 102. 

The terrain detection system 104 is used to detect obstacles 
around the vehicle 102 (e.g., rocks, sudden drop-offs, trees, 
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and so forth) and/or detect a path for the vehicle 102 to travel 
(e.g., detecta road). The terrain detection system 104 receives 
and analyzes input from the terrain sensors 160 to identify 
obstacles and a path for the vehicle 102 to follow. Terrain data 
from the various terrain sensors 160 is merged into a single 
database that is used to plan the path of the vehicle 102. 

The terrain detection system 104 includes a first LIDAR 
system 103, a second LIDAR system 105, and a vision system 
107. The first LIDAR system 103 receives input from two 
single-plane LIDAR sensors 162, 164 coupled to the vehicle 
102. The second LIDAR system 105 receives input from two 
multi-plane LIDAR sensors 166 coupled to the vehicle 102. 
The vision system 107 receives input from three cameras 168, 
170, 172 coupled to the front of the vehicle 102. It should be 
appreciated that the number, configuration, and types of ter- 
rain sensors used to provide input to the terrain detection 
system 104 may be varied widely. For example, the terrain 
detection system may include a radar system, or may include 
additional LIDAR systems, or may include a rotating array of 
LIDAR sensors to provide a 360? view of the surrounding 
terrain. Likewise, the terrain sensors 160 may include a radar 
antenna, various cameras (still and/or video cameras), and 
multiple LIDAR sensors (e.g., single plane or multi-plane 
LIDAR systems). Also, the vehicle 102 may include any of 
these sensors in any configuration coupled to the rear of the 
vehicle 102 to facilitate operation of the vehicle 102 in 
reverse. 

The terrain sensors 160 may be mounted to the vehicle 102 
using any suitable mounting structure. For example, as shown 
in FIG. 12, the terrain sensors 160 may be mounted to the 
vehicle using telescopically adjustable mounts 157 that move 
in and out via a threaded engagement system. The adjustable 
mounts 157 allow for precise adjustment ofthe terrain sensors 
160 but also keep the terrain sensors 160 in a fixed position 
after the adjustment has been performed. The adjustable 
mounts 157 may be used to hold any ofthe terrain sensors 160 
and/or position sensors 150 in place. The terrain sensors 160 
and/or position sensors 150 may be coupled to a roll bar or roll 
cage 180 of the vehicle. The roll bar may also serve as a 
protective conduit for the sensor communication wires and 
power wires to travel. 

As shown in FIGS. 3-4, the LIDAR sensors 162, 164 are 
coupled to the outside edges ofthe roll bar 180 at the front of 
the vehicle 102. The LIDAR sensors 162, 164 are positioned 
so that they point about 10 degrees down and about 25 degrees 
outward to the side ofthe vehicle 102. Positioning the LIDAR 
sensors 162, 164 in this fashion provides good coverage ofthe 
terrain to the front and the side ofthe vehicle 102. This allows 
the vehicle 102 to make tight turns while still having some 
information regarding the terrain in the direction that the 
vehicle 102 is turning. It should be appreciated that the 
LIDAR sensors 162, 164 may be positioned at any suitable 
position to provide the most useful information to the vehicle 
control system 100. To some extent, the orientation of the 
LIDAR sensors 162, 164 depends onthetype of vehicle being 
used (e.g., taller vehicles may require the LIDAR sensors to 
be pointed down more). The LIDAR sensors 162, 164 may be 
configured to be positioned about 0 to 20 degrees down and 
about 0 to 90 degrees outward to the side of the vehicle 102. 
Also, the LIDAR sensors 162, 164 may be configured to scan 
a 100 degree scan area with a 1 degree resolution. Suitable 
LIDAR sensors may be obtained from SICK, having an office 
in Minneapolis, Minn., as model number SICK LMS-291. 

The first LIDAR system 103 may be configured to detect 
positive obstacles (e.g., obstacles that protrude upward) and 
negative obstacles (e.g., road edges, cliffs, etc.). The position- 
ing of the LIDAR sensors 162, 164 should allow for detection 
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of both positive and negative obstacles. The first LIDAR 
system 103 processes the scan data from the LIDAR sensors 
162, 164 to identify positive and negative obstacles. Positive 
obstacles may be detected by translating the raw scan data 
from a sensor level coordinate framework to a local level 
coordinate framework and comparing the relative heights of 
neighboring scan points. A history of scan data can be used to 
effectively map the surface in front of the vehicle 102. An 
obstacle is identified if the surface as detected by the LIDAR 
sensors 162, 164 protrudes upward above a threshold amount. 
The detection threshold may be set at a minimum obstacle 
height based on the capabilities of the vehicle 102. For 
example, the vehicle 102 as shown in FIG. 1 may be config- 
ured to have a detection threshold of about eighteen inches 
since the vehicle 102 can drive over any obstacle that is under 
eighteen inches. A convex hull algorithm may be used to 
define the outermost edge of the obstacle. Once the obstacle 
and its outermost edges have been identified, the vehicle 
control system 100 can store the obstacle in the database. 

The first LIDAR system 103 may detect negative obstacles 
in amanner that is similar to detecting positive obstacles. The 
data map of the surface in front of the vehicle 102 may be 
searched using an algorithm that identifies any negative 
height discontinuities. Any identified negative height discon- 
tinuities are evaluated further to determine if the edge of the 
discontinuity is a continuation of a previously detected edge. 

It should be appreciated that many aspects related to the 
first LIDAR system 103 and the single-plane LIDAR sensors 
162, 164 may be modified in a variety of ways. For example, 
instead of using single-plane LIDAR sensors, multi-plane 
LIDAR sensors may be used. Also, more or less than two 
LIDAR sensors 162, 164 may be used (e.g., one or more 
additional LIDAR sensors may be coupled to the rear of the 
vehicle 102 to assist with backing the vehicle 102). The 
LIDAR sensors 162, 164 may be coupled to the vehicle at any 
suitable location such as the cab, fender, and so forth. 

Referring to FIGS. 3, 5, and 7, the multi-plane LIDAR 
sensors 166 used with the second LIDAR system 105 are 
coupled to the vehicle 102 near the bumper 182. Each of the 
LIDAR sensors 166 are four plane scanner devices that are 
used primarily for positive obstacle detection at long and 
close range. Each LIDAR sensor 166 is positioned horizon- 
tally so that two planes scan toward the ground and two planes 
scan toward the sky. Desirably, the LIDAR sensors 166 each 
have a range of about 80 meters, a resolution of about 0.25 
degrees, and a 170 degree scan area. The large scan area 
allows the LIDAR sensors 166 to be used to identify obstacles 
around upcoming turns. 

As shown in FIG. 1, the vehicle 102 may be configured to 
include only a single LIDAR sensor 166. However, it may be 
desirable to include at least two LIDAR sensors 166 so that in 
case one of the LIDAR sensors 166 is damaged or otherwise 
unable to be used, the remaining LIDAR sensor 166 can still 
provide information to the vehicle control system 100. 

The second LIDAR system 105 is configured to receive 
scan data from the LIDAR sensors 166 and identify obstacles 
by searching for large positive slopes in the scan data. The 
obstacle detection algorithm used to process the scan data in 
the first LIDAR system 103 is similar to the obstacle detection 
algorithm used to process the scan data in the second LIDAR 
system 105. In one embodiment, the same algorithm may be 
used to process scan data from both sources. The second 
LIDAR system 105 may be configured to translate the scan 
data from the coordinate frame of the particular LIDAR sen- 
sor 166 to a local level coordinate frame and stored in the 
database. Since the LIDAR sensors 166 are multi-plane 
devices, the scan data from each plane may be compared to 
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each other to aid in detecting obstacles and false positives. 
Since the scan data has been translated into a local level 
coordinate frame, the position of obstacles identified in the 
scan data from the LIDAR sensors 166 can be compared to the 
position of obstacles identified using the other terrain sensors 
160 and to the position of the vehicle 102. 

The LIDAR sensors 162, 164 and the LIDAR sensors 166 
may be configured to scan the same area or different areas in 
front of the vehicle 102. In one embodiment, the LIDAR 
sensors 162, 164, 166 are used to scan overlapping areas in 
front of and to the side of the vehicle 102. In another embodi- 
ment, the LIDAR sensors 162, 164 may be configured to scan 
for obstacles that are 0 to 20 meters in front ofthe vehicle 102 
and the LIDAR sensors 166 may be configured to scan for 
obstacles that are 20 to 60 meters in front of the vehicle 102. 
Obstacles detected by the LIDAR sensors 162, 164, 166 may 
be put into an obstacle database that the vehicle control sys- 
tem 100 uses to plan the path ofthe vehicle 102. 

The vision system 107 may be based on a multi stereo- 
scopic vision system. The vision system 107 receives input 
from the cameras 168, 170, 172, which are coupled to a 
horizontal portion ofthe roll bar 180. The cameras 168, 170, 
172 are coupled to the vehicle 102 in a secure manner to 
prevent the cameras from becoming dislodged during opera- 
tion of the vehicle 102. This may especially be a problem 
when the vehicle 102 operates in an off-road environment. 

The cameras 168, 170, 172 are coupled to the vehicle 100 
so that they are asymmetrically spaced apart from each other. 
As shown in FIG. 4, the cameras 168, 172 are positioned 
about 1.5 meters apart from each other while the camera 170 
is positioned about 0.5 meters from the camera 168. In one 
embodiment, the cameras 168, 170, 172 may be video cam- 
eras that transmit images using ordinary light to the vision 
system 107. In another embodiment, the cameras 168, 170, 
172 may be still cameras that periodically take pictures at the 
same time. The pictures may be transmitted to the vehicle 
control system 100 where the pictures are processed to iden- 
tify obstacles and/or find a path for the vehicle 102. In yet 
another embodiment, the cameras 168, 170, 172 may be 
infrared cameras (either video or still). In yet another embodi- 
ment, the cameras 168, 170, 172 may be video cameras 
capable of providing video streams at 10 Hz (640x480, color 
with Bayer pattern). 

The orientation of the cameras 168, 170, 172 may be cali- 
brated using a graduated grid. This may be done to fix the 
three degrees of freedom specifying the cameras orientation 
to known values. In one embodiment, the yaw and roll angles 
ofthe cameras 168, 170, 172 are fixed at zero. This may help 
to reduce the processing requirements of the images taken by 
the cameras 168, 170, 172. The pitch angle of the cameras 
168, 170, 172 may be chosen so that the cameras 168, 170, 
172 frame a small portion over the horizon (to limit direct 
sunlight) and frame the terrain at about four meters from the 
vehicle 102. FIGS. 18-23 show images taken with the pitch of 
the cameras 168, 170, 172 being adjusted in this manner. 

It should be appreciated that the cameras 168, 170, 172 
may be coupled to the vehicle 102 in any suitable fashion and 
location. For example, additional cameras may be coupled to 
the rear and side of the vehicle 102 to facilitate backing and 
turning of the vehicle 102. Also, the pitch, yaw, and roll of the 
cameras 168, 170, 172 may be adjusted in any suitable fash- 
ion. 

As shown in FIG. 2, the cameras 168, 170, 172 send three 
video streams to the vision system 107 via fire wire connec- 
tions. The vision system 107 selects which stereo pair of 
video streams to use depending on the speed of the vehicle 
102. By selecting different pairs of cameras, the vision system 
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107 can obtain images from stereo pairs of cameras having 
different baselines (1.е., the distance between the cameras). It 
is desirable to have the ability to vary the baseline because 
when the baseline is wider, the vision system 107 is able to 
more accurately determine the distance to an object, but as the 
baseline increases, it makes it more difficult to stereo match 
objects (especially close objects) in the images because the 
objects are viewed at very different angles. Therefore, the 
vision system 107 is configured to receive the video streams 
from cameras 168, 172 when the vehicle 102 is traveling at 
high speeds so that the vision system 107 can accurately 
identify and measure the distance of obstacles that are a long 
distance away. When the vehicle 102 is traveling at medium 
speeds and obstacles are not approaching as quickly, the 
vision system 107 is configured to receive the video streams 
from cameras 170, 172. Likewise, when the vehicle 102 is 
traveling at slow speeds, the vision system 107 is configured 
to receive the video streams from cameras 168, 170 in that 
order. This configuration allows the vision system 107 to 
obtain images from cameras having a wide baseline when the 
vehicle 102 is moving at high speeds, obtain images from 
cameras having a medium baseline when the vehicle 102 is 
moving at medium speeds, and obtain images from cameras 
having a short baseline when the vehicle 102 is moving at 
slow speeds. 

The use of at least three cameras 168, 170, 172 fixed in 
position provides a mechanically robust system that is 
capable of providing images from different combinations of 
the cameras 168, 170, 172. In other embodiments, the vision 
system 107 may be configured to receive input from two 
cameras where at least one of the cameras rotates and tilts as 
the speed of the vehicle 102 changes instead of using the three 
camera configuration. The moving cameras provide more 
accurate distance estimation and better object matching ver- 
sus a two camera configuration where the cameras do not 
move. However, the configurations that use at least three fixed 
cameras are preferred since they allow the vision system 107 
to receive different combinations of video streams from cam- 
eras having different baselines and they do not rely on move- 
ment of the cameras—an area that is more prone to failure 
and/or that may introduce variations in the video streams. 

It should be appreciated that the vehicle control system 100 
may include additional LIDAR systems or may combine the 
first LIDAR system 103 and the second LIDAR system 105 
into a single LIDAR system. It should also be appreciated that 
the data from the LIDAR and vision systems may be pro- 
cessed by a single processor and/or single software program. 
It should also be appreciated that in other embodiments, the 
data from each LIDAR system and vision system may be 
initially processed using a separate system. The output from 
each system may then be compared to create a database of 
obstacles. It should also be appreciated that each of these 
systems may be provided as a separate module that may 
optionally be included with the vehicle control system 100. 
Providing each system as a separate module may allow for 
greater flexibility in selecting the components of the vehicle 
control system 100 to use with a particular type of vehicle. 

The vision system 107 may be used to detect obstacles and 
detect a path of travel for the vehicle 102. The images pro- 
vided by the video streams are subjected to V-disparity image 
analysis to determine the average terrain slope in front of the 
vehicle 102. Slope information is then used for both obstacle 
detection and path detection. Any significant deviation from 
the average smooth slope detected previously is identified as 
an obstacle. The exact location of the obstacle is then obtained 
via stereo triangulation between the two views of the obstacle. 
This method provides a fairly precise position of the obstacle. 
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However, the vehicle control system 100 may also compare 
the position of the obstacle as determined by the vision sys- 
tem 107 with the position ofthe obstacle as determined by the 
first LIDAR system 103 and/or the second LIDAR system 
105 to provide an even more accurate position ofthe obstacle. 

This method allows the vision system 107 to detect thin 
vertical posts such as fence posts. Although the vision system 
107 may be capable of detecting small obstacles, the vision 
system 107 may be tuned with high thresholds to reduce the 
number of false positives. In other words, the capability of 
detecting small obstacles may be traded for a higher robust- 
ness of detection. 

FIGS. 16-19 show pictures highlighting obstacles detected 
by the vision system 107. Each of FIGS. 16-19 includes two 
pictures—the one on the left is the image before the obstacles 
are detected and the one on the right is the same image with 
the obstacles highlighted. The different colors of the 
obstacles represent the distance to the obstacle from the 
vehicle 102 (orange are the closest, red the next closest, and 
purple are the farthest). As shown in FIGS. 16-19, posts and 
thin poles are detected by the vision system 107. 

The vision system 107 also uses image disparity analysis as 
the initial step in detecting a path for the vehicle 102 to travel. 
Image disparity analysis is used to compute the area in front 
of the vehicle 102 which features a smooth slope, which is 
known as the free-space. Free space is obtained using stan- 
dard image warping to localize deviations from a smooth road 
surface. FIGS. 20A-20C show the left image, the warped right 
image, and the matching cluster representing free space. 

The vision system 107 combines the free space with other 
features ofthe images such as similarity in texture, similarity 
in color, and/or similarity in shape to construct a representa- 
tion of a path to follow. The path detection algorithm of the 
vision system 107 is also configured to indicate whether the 
path is straight so that the vehicle 102 may increase speed. 
When a curved path is present, the vehicle 102 is configured 
to slow down. 

The cameras 168, 170, 172 may be configured to compen- 
sate for different light levels using automatic gain control, 
which allows the cameras 168, 170, 172 to sense the environ- 
ment including direct sunlight into the cameras 168, 170, 172. 
FIGS. 21-23 show examples of images captured with and 
without gain control. FIGS. 22-23 show the result of obstacle 
detection by the vision system 107 in bad light conditions. As 
shown in FIGS. 22-23, obstacles may still be detected up until 
they enter the oversaturated area. The cameras 168, 170, 172 
may also be provided with sun shades 186, which reduce the 
amount of direct sunlight that hits the cameras 168, 170, 172 
to avoid oversaturation and reflections due to dirty glass. The 
vehicle 102 may also include shields or reflectors 188 that 
reflect sunlight away from the area of the cameras 168, 170, 
172 and the LIDAR sensors 162, 164 to prevent these devices 
from becoming too hot. In one embodiment, the shield 188 is 
a reflective diamond plating material (e.g., aluminum or 
steel). Additional shields 189 may be provided in the vicinity 
ofthe LIDAR sensors 166. 

The vehicle 102 may include a cleaning system that is 
configured to clean the lenses on the cameras 168, 170, 172 
and/or the lenses of the LIDAR sensors 162, 164, 166. The 
cleaning system may include one or more nozzles that are 
aimed at the lenses. The nozzles may be configured to dis- 
pense any suitable gas or liquid onto the lens to clean it. In one 
embodiment, the cleaning system may be configured to 
include an air knife that directs high velocity bursts of air onto 
the lenses to remove any dust or other debris. The use of air 
may be desirable since liquids may attract dust and dirt and/or 
leave spots on the lenses. In another embodiment, the clean- 
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ing system may be configured to direct water or another liquid 
(e.g., windshield washer fluid, and so forth) towards the lens. 
The cleaning system may then direct air over the lens to 
quickly dry the lens and remove the water. The air may be 
supplied by a compressed air system that is included as part of 
the vehicle 102. In another embodiment, the cleaning system 
may comprise a cleaning controller, one or more valves (e.g., 
anarray of valves), and a washer tank. The cleaning controller 
may be configured to electronically control the valves to 
direct liquid and/or air through the nozzles and onto the 
lenses. 

The vehicle management system 106 includes a path plan- 
ning module 200, a vehicle control module 202, a navigation 
module 204, and an information database 206. The vehicle 
management system 106 is configured to receive input from 
the terrain detection system 104 and control the vehicle 102 to 
reach a desired location. The vehicle management system 106 
may also include a modeling and simulation system that is 
used to determine the capabilities of the vehicle 102 under 
different circumstances. The modeling and simulation system 
may be similar to that described in U.S. Prov. Pat. App. No. 
60/723,363, entitled “Vehicle Control System and Method,” 
filed on Oct. 4, 2005, which is hereby incorporated herein by 
reference in its entirety. 

The navigation module 204 may be configured to compute 
the present position of the vehicle 102 and perform dead 
reckoning navigation. Dead reckoning navigation may be 
needed when the vehicle 102 is unable to receive GPS signals 
(e.g., when the vehicle 102 is in a tunnel, etc.). The navigation 
module 204 may also include navigation related information 
such as maps (e.g., road maps, topographical maps, and the 
like), aerial photographs, a list of GPS waypoints, and the 
like. This information may be utilized by the navigation mod- 
ule 204 to drive the vehicle 102 to its destination. In one 
embodiment, each mission may be planned out by setting a 
plurality of GPS waypoints and a preferred path of travel that 
the vehicle 102 should take until it reaches its final destina- 
tion. The GPS waypoints and preferred path may be loaded 
into the navigation module 204 prior to the vehicle 102 
embarking on the mission. During the mission, the vehicle 
control system 100 is configured to drive the vehicle to suc- 
cessive waypoints while trying to stay on the preferred path as 
much as possible while still avoiding any obstacles that may 
present themselves. 

The navigation module 204 receives input from the posi- 
tion sensors 150. In the embodiment shown in FIG. 2, the 
position sensors 150 include dual GPS/inertial measurement 
units (IMU) 152 and a differential GPS unit 154. The GPS/ 
IMU units 152 may include a six-axis inertial measurement 
system which includes three angular rate sensors (gyros) and 
three accelerometers. The three accelerometers are all 
mounted at 90 degrees to each other, and the three angular rate 
sensors are mounted at the same angles as their respective 
accelerometers. Together they give full 3D motion measure- 
ment of the vehicle 102. The GPS/IMU units 152 are also 
capable of receiving GPS signals directly from GPS satellites 
and differential GPS information from the differential GPS 
unit 154. The information provided to the navigation module 
204 is used to chart the course of the vehicle 102 at a high 
level. The vehicle 102 attempts to follow the course as much 
as possible while avoiding obstacles and attempting to stay on 
a road or other path. 

Input from the inertial measurement system may be used 
by the vehicle control system 100 to perform dead reckoning 
navigation. The inertial position information provided to the 
vehicle control system 100 may be supplemented with other 
data such as wheel speed and time traveled (e.g., to determine 
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the distance traveled during dead reckoning navigation), 
wheel speed and steering angle (e.g., to determine the direc- 
tion the vehicle 102 is going during dead reckoning naviga- 
tion), or any other information that may be useful to deter- 
mine the position of the vehicle 102. Also, the inertial position 
information may be used to measure the pitch and roll of the 
vehicle 102. In one embodiment, the vehicle control system 
100 may be configured to perform dead reckoning navigation 
using sensed wheel speed and wheel angle for those situations 
where the GPS and IMU units are not operational. The GPS/ 
IMU units 152 may be positioned on the approximate center- 
line of the vehicle 102 (e.g., in the floor of the cab). In one 
embodiment, the GPS/IMU units 152 may be obtained from 
Oxford Technical Solutions, Ltd., having an office in Dear- 
born, Mich., as part number OXTS RT3100. 

In the embodiment shown in FIG. 2, the vehicle 102 
includes two GPS/IMU units 152. Two of the units 152 may 
be provided so that the position information (both GPS and 
inertial) received from each unit can be averaged to provide a 
more accurate and robust calculation of the position of the 
vehicle 102. Also, if one of the GPS/IMU units 152 fails, the 
remaining GPS/IMU unit 152 could still be used to provide 
position information to the vehicle control system 100. The 
vehicle control system 100 may be able to determine that one 
of the GPS/IMU units 152 has failed when, for example, the 
unit 152 stops providing input or the input is so different from 
the position information provided by the other position sen- 
sors 150 and terrain sensors 160 that the input can no longer 
be trusted. If the vehicle control system 100 determines that 
one of the GPS/IMU units 152 has failed, the vehicle control 
system 100 uses the remaining GPS/IMU unit 152. 

The differential GPS unit 154 is configured to receive 
differential corrections and send them to the GPS/IMU units 
152. The differential GPS unit 154 may be configured to 
receive Beacon, SBS/WAAS, and/or Omnistar corrections, as 
well as any other suitable corrections. It should be appreci- 
ated that the combination of GPS units may be varied widely. 
For example, the GPS/IMU units 152 may be capable of 
receiving the differential corrections directly without using a 
separate receiver. Likewise, the GPS/IMU units 152 may be 
split into two units so that the GPS unit is separate from the 
IMU unit. 

The vehicle control module 202 is configured to interface 
with the vehicle subsystems 110 to allow the vehicle compo- 
nents to be controlled by the vehicle management system 106. 
As shown in FIG. 2, the vehicle subsystems 110 may include 
a steering system 112, a braking system 116, a throttle system 
120, and a transmission system 124. 

The steering system 112 may include a servo drive 113 and 
a corresponding servomotor 114 that is in communication 
with the servo drive 113. The servo drive 113 may be config- 
ured to receive and process position commands from the 
vehicle control system 100 and provide corresponding out- 
puts (e.g., an analog voltage or current) to the servomotor 
114. In response, the servomotor 114 may drive one or more 
associated wheels of the vehicle 102 to a desired steering 
angle. The steering system 112 may also include one or more 
wheel angle sensors that are used to determine the steering 
angle of a wheel of the vehicle 102 and provide feedback to 
the vehicle control system 100 as to the current steering angle 
of the wheel. In another embodiment, the servomotor 114 
may include an integrated high-resolution encoder that is 
used to control the steering angle of the vehicle 102. It should 
also be appreciated that the encoder may be coupled to the 
steering system 112 at any location that is capable of provid- 
ing a reliable and accurate measurement of the steering angle 
of the wheels (e.g., steering column, etc.). 
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In one embodiment, the servomotor 114 is coupled directly 
to the steering gearbox of the vehicle 102. The steering gear- 
box may be configured to have dual inputs so that the servo- 
motor 114 and the hand wheel can both be connected to the 
steering gear. This allows steering control of the vehicle to be 
switched between manual mode (operated by a driver) and 
autonomous mode without disassembling the steering sys- 
tem. 

The steering system 112 may also be an all-wheel steering 
system, or in the case of 6x6 wheeled vehicles, 8x8 wheeled 
vehicles and so on, the steering system 112 may be configured 
to steer one, two, three, four, or more sets of wheels. The 
steering system 112 may be configured similarly to any of the 
steering systems described in U.S. Pat. No. 6,882,917 (incor- 
porated by reference herein previously). 

The braking system 116 may include, for example, an 
anti-lock brake system including an anti-lock brake control 
system and anti-lock brakes coupled to one or more wheels of 
the vehicle 102. The braking system 116 may be configured to 
receive braking commands, such as a “brake ON” command, 
from the vehicle control system 100. In response, the braking 
system 116 may apply braking pressure to one or more wheels 
of the vehicle 102 to, for example, reduce the speed of the 
vehicle 102 or bring the vehicle 102 to a stop. According to an 
exemplary embodiment, braking pressure for the braking sys- 
tem 116 may be provided by a compressed-air system. The 
compressed-air system may include, for example, an air com- 
pressor coupled to the power train ofthe vehicle 102 (such as 
via gears, v-belt, etc. to convey rotational mechanical energy 
to the air compressor to be converted to air pressure), and one 
or more air pressure tanks for storing compressed air from the 
air compressor. 

It should be noted that for purposes of this disclosure, the 
term “coupled” means the joining of two members directly or 
indirectly to one another. Such joining may be stationary in 
nature or movable in nature. Such joining may be achieved 
with the two members or the two members and any additional 
intermediate members being integrally formed as a single 
unitary body with one another or with the two members or the 
two members and any additional intermediate member being 
attached to one another. Such joining may be permanent in 
nature or alternatively may be removable or releasable in 
nature. 

As shown in FIG. 2, the vehicle control system 100 may be 
configured to communicate with the braking system 116 via 
Ethernet to a proportional voltage to pressure valve. The 
braking system 116 may be configured to apply the service 
brakes in an amount that is proportional to the deviation 
between the actual speed of the vehicle 102 and the desired 
speed of the vehicle 102. It should be appreciated, however, 
that numerous other configurations may also be used to con- 
trol the braking system 116. For example, the vehicle control 
system 100 may be configured to control the braking system 
116 using more mechanical components such as a hydraulic 
actuator that operates the brakes through the system. 

The braking system 116 may also include an engine brake. 
The braking system 116 in combination with the throttle 
system 120 may be used to control the speed of the vehicle 
102. Typically, the speed of the vehicle 102 is controlled 
solely using the throttle. However, the service brakes and the 
engine brake may be used to control the speed of the vehicle 
102 on steep grades and/or to manage the speed that the 
vehicle 102 decelerates. 

The throttle system 120 may include an engine control 
module and a throttle actuator. The engine control module 
may be configured to receive and process throttle commands 
from the vehicle control system 100 and provide correspond- 
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ing outputs (e.g., an analog voltage or current) to the throttle 
actuator. In response, the throttle actuator may adjust the 
rotational speed of an engine coupled to the drive train of the 
vehicle 102 to achieve a desired speed of the vehicle 102. The 
engine control module may be configured to determine the 
speed of the vehicle 102 (e.g., using a tachometer feedback 
signal) and provide feedback to the vehicle control system 
100 as to the current speed of the vehicle 102. In other 
embodiments, the speed of the vehicle 102 may be deter- 
mined using feedback from the transmission system 124 or by 
measuring the rotation of a drive shaft, or some other com- 
ponent of the power train of the vehicle 102. In one embodi- 
ment, the vehicle control system 100 and/or the engine con- 
trol module may use a pulse width modulated (PWM) signal 
to provide very precise control of the throttle. 

The throttle system 120 may be configured so that reset 
conditions to the throttle position are provided for transmis- 
sion up shift and down shift and to activate the engine brake. 
Since the engine brakes are typically activated when the 
engine is idling, throttle position overrides are used whenever 
the engine brakes are active. Throttle position faders are used 
to reactivate the throttle position control when the engine 
brake is disabled. 

The transmission system 124 may include the engine con- 
trol module and a transmission mechanically coupled to the 
engine of the vehicle 102. In combination, the transmission 
system 124 and the engine of the vehicle 102 form the power 
train of the vehicle 102. The engine control module may be 
configured to receive and process gear change commands 
from the vehicle control system 100 and provide correspond- 
ing outputs (e.g., an analog voltage or current) to the trans- 
mission system 124. In response, the transmission may 
change gears to achieve a desired rotational speed of the 
power train of the vehicle 102. In another embodiment, the 
transmission system 124 may include a transmission control 
module that is separate from the engine control module. The 
transmission control module may be in communication with 
the engine control module so that information required for 
shifting gears and so forth can be communicated between the 
transmission control module and the engine control module. 

As shown in FIG. 2, the vehicle subsystems 110 may also 
include an emergency stop functionality 127. The emergency 
stop 127 may beany suitable switch configured to provide, for 
example, stop, pause, or run commandis to the vehicle control 
system 100 to either stop, temporarily interrupt, or initiate the 
operation of the vehicle 102. In one embodiment, the emer- 
gency stop functionality 127 may cause a full brake command 
to be provided to the braking system 116. The full brake 
command results in the service brakes being applied in full to 
stop the vehicle 102 as quickly as possible. 

The vehicle subsystems 110 may also include an odometer 
128. The odometer 128 may be used to provide information 
regarding the movement of the vehicle 102 (e.g., distance 
traveled) to the vehicle control system 100 in order to enhance 
the ability of the vehicle control system 100 to determine the 
position or movement of the vehicle 102. The odometer may 
also be used by the vehicle control system 100 to perform 
dead reckoning navigation. 

The path planning module 200 is configured to determine 
the desired trajectory of the vehicle 102 and provide that 
trajectory to the vehicle management system 106. The trajec- 
tory includes a desired path along the ground as well as the 
desired speeds and boundary area. The desired trajectory may 
be determined using path and speed constraints defined 
before the vehicle 102 begins operating autonomously (e.g., 
pre-mission planning stage). As mentioned above in connec- 
tion with the navigation module 204, the route that the vehicle 
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102 follows may be defined beforehand and loaded into the 
vehicle control system 100. For example, the route may be 
defined in terms of a series of waypoints that define a path 
along the ground, a path boundary, and/or speed constraints 
for each section of the path. The path planning module 200 
provides reactive path corrections to this nominal path to 
account for current conditions, such as vehicle dynamic lim- 
its, obstacles (positive obstacles suchas large rocks, trees, etc. 
and negative obstacles such as road edges, drop-offs, and 
cliffs), terrain grade, and so forth. 

In one embodiment, the path planning module 200 may be 
configured to use a tree algorithm that branches from the base 
at the current waypoint. The tree build function may be con- 
strained by the pre-defined path boundary and/or speed so 
that the resulting tree is not too large to process. Branches of 
the tree are computed using a model of the vehicle 102 (e.g., 
steering system and vehicle dynamics) to ensure that the 
candidate paths are drivable. One potentially suitable model 
is described in U.S. Prov. Pat. App. No. 60/723,363, entitled 
“Vehicle Control System and Method,” filed on Oct. 4, 2005, 
which is hereby incorporated herein by reference in its 
entirety. The tree algorithm may be derived from the Rapidly- 
exploring Random Tree (RRT) path planner where the growth 
of the tree is limited to a fixed number of branches (levels). 

The tree represents a series of candidate paths for the 
vehicle 102 to travel along. The vehicle control system 100 
selects a path from the various candidate paths based on a 
scoring algorithm that considers distance from the route cen- 
terline, path curvature, obstacle avoidance, boundary area 
constraints, and so forth. The path planning module 200 may 
generate at least 100 candidate paths, at least 1000 candidate 
paths, or at least about 2000 candidate paths each planning 
cycle. Typically, the path planning module 200 performs a 
path analysis each time a waypoint is reached. The path 
planning module 200 may also be configured to adjust the 
maximum speed of the vehicle 102 to account for path geom- 
etry and current conditions. The initial desired speed may be 
set to the maximum determined during the pre-mission plan- 
ning stage. The speed management component of the path 
planning module 200 may be configured to adjust the initial 
set speed depending on conditions. In one embodiment, the 
speed management component of the path planning module 
200 may be configured to reduce the maximum speed of each 
section of the path depending on the current conditions. 

In another embodiment, the path planning module 200 may 
be configured to continually calculate and adjust the path 
while the vehicle 102 is traveling (e.g., real-time path plan- 
ning). In yet another embodiment, the path planning module 
200 may be configured to reevaluate the path if the vehicle 
102 encounters a situation that prevents the vehicle 102 from 
continuing on the original path (e.g., a washed out road, fallen 
tree, rock slide, etc.). In this situation, the path planning 
module 200 may be configured to determine the best way for 
the vehicle 102 to proceed to the waypoint. The path planning 
module 200 may include or have access to some of the same 
information that may be included in the navigation module 
204 such as topographical maps, road maps, aerial or satellite 
images, and so forth, to allow the path planning module 200 
to determine the best path for the vehicle 102. 

FIG. 13 shows a graphical representation of a path gener- 
ated by the path planning module 200. In FIG. 13, the vehicle 
102 is represented by the long rectangular box. The sequence 
of boxes 208 represents the path that is loaded into the vehicle 
management system 106 before the vehicle 102 begins oper- 
ating autonomously. The numbers identified by the reference 
numeral 210 represent the real-time path generated by the 
path planning module 200. FIG. 13 also shows obstacles 212 
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detected by the terrain detection system 104. As shown in 
FIG. 13, thepath 210 ofthe vehicle 102 is adjusted to the right 
of the center of the pre-loaded path to avoid obstacles in the 
turn. The data boxes (e.g., control data, command response, 
state data, path data) in FIG. 13 show data associated with the 
path, vehicle state, and control state of the vehicle 102. 

FIG. 14 is an illustration ofthe path 224 (the path 224 is the 
sequence of numbers shown in FIG. 14) generated by the path 
planning module 200 which shows obstacles 222 detected 
using the first LIDAR system 103. Similarly, FIG. 15 is an 
illustration of the path 226 (the path 226 is the sequence of 
numbers shown in FIG. 15) generated by the path planning 
module 200 which shows obstacles 228 detected using the 
second LIDAR system 105. 

The vehicle management system 106 controls the move- 
ment of the vehicle 102 along the path by providing wheel 
angle commands to the steering system 112. The wheel angle 
commands move the vehicle 102 along the path defined by the 
path planning module 200. This may be accomplished by 
computing deviations from the desired path and converting 
the deviations into steer angle commands that are sent to the 
steering servo. 

The vehicle management system 106 may include a cap- 
ture steering mode and a track steering mode. The capture 
steering mode refers to the steering mode that is used based on 
the pre-loaded path data. The track steering mode refers to the 
steering mode that is used during normal operation. Capture 
and track steering modes may be automatically selected 
based on current conditions. 

The capture steering mode uses course error as the control 
parameter. The vehicle management system 106 creates a 
steer angle command that aligns the ground track of the 
vehicle with the direct bearing to the next waypoint. This type 
of control may be referred to as homing control. Capture 
steering mode is typically only used when the vehicle 102 first 
begins autonomous operation. 

The track steering mode uses linear cross track deviation 
and cross track deviation rate to align the vehicle's path along 
the ground with the waypoint course. Track angle error and 
steer angle command limiters may be used to limit the com- 
manded steer angle to values that are achievable by the 
vehicle 102. The command limiters incorporate vehicle 
dynamic limits with margins built in to ensure that the vehicle 
does not get into an unsafe condition. This may also mean that 
the vehicle 102 operates at levels below its maximum 
dynamic capability when in autonomous mode. The vehicle 
management system 106 may also anticipate turns so that the 
transition between traveling between waypoints may be 
accomplished without overshoots. 

The vehicle management system 106 is configured to 
monitor and dynamically change to conditions using infor- 
mation from the path planning module 200, the navigation 
module 204, the vehicle control module 202, and the infor- 
mation database 206. The vehicle management system 106 
may be configured to handle a number of situations that may 
arise during autonomous operation. When the situation 
occurs, the vehicle management system 106 is configured to 
proceed through certain operations to address the situation. 

Examples of situations that may arise include: 

(1) transition to and from e-stop state—when the e-stop is 
activated, the vehicle management system 106 is configured 
to stop the vehicle 102. When the e-stop is deactivated, the 
vehicle management system 106 is configured to initiate nor- 
mal operation. 

(2) no valid path ahead—the vehicle 102 is commanded to 
stop and wait for the vehicle management system 106 to 
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obtain a valid path. If no valid path is found, the vehicle 
management system 106 will back the vehicle 102 up and try 
again. 

(3) obstacle detected behind vehicle while backing 
up—the vehicle management system 106 stops the vehicle 
102 and initiates normal operation of the vehicle 102 to finda 
valid path ahead. 

(4) a large course change requiring a back up maneuver— 
the vehicle management system 106 operates the vehicle 102 
around a three-point turn. 

(5) narrow tunnel—the vehicle management system 106 
guides the vehicle 102 through the tunnel primarily using the 
LIDAR systems 103, 105. 

(6) stuck between obstacles—the vehicle management sys- 
tem 106 implements the stuck behavior when the vehicle 102 
cannot make progress along the route because it is continually 
going back and forth so that it is stuck between obstacles. The 
vehicle management system 106 positions the vehicle 102 at 
different angles to search for a valid path. If no valid path is 
found, then the vehicle management system 106 ignores low 
confidence obstacles in an attempt to eliminate low confi- 
dence obstacles. The last resort is to go forward along the 
pre-loaded route ignoring all obstacles. 

The vehicle control system 100 may also be configured to 
detect a collision involving the vehicle 102. Upon detection of 
a collision, the vehicle control system 100 may be configured 
to run a diagnostic check on each input and output device that 
is part of the vehicle control system 100 to determine if all of 
them are online and functional. 

The functionality of the vehicle management system 106 
may be similar to the Distributed Architecture for Mobile 
Navigation (DAMN) where the vehicle ordinarily follows the 
path generated by the path planning module 200 but the path 
can be overwritten depending on the current conditions. How- 
ever, in some embodiments, the DAMN architecture may be 
modified to use a rules-based decision logic (instead of a 
complex voting scheme) to determine when the path should 
be overwritten. The use of rules-based decision logic may be 
more deterministic and robust than other solutions, which 
may render itself more useful in fleet applications. 

The system manager 108 is the module of the vehicle 
control system 100 that allows a user to interface with the 
vehicle control system 100 to execute control commands and 
review the status of various systems and devices. The system 
manager 108 is also configured to continuously monitor the 
status of the various applications included in the vehicle con- 
trol system 100. Ifthe system manager 108 determines that an 
application is no longer functioning normally, the system 
manager 108 is configured to stop and restart the application 
to return normal functionality. The vehicle control system 
100 may be configured so that ifthe system manager 108 fails, 
the vehicle control system 100 can continue operating. 

In the context of a convoy of vehicles 102, the first vehicle 
102 in the convoy may be configured to operate autono- 
mously. The remainder of the vehicles may operate autono- 
mously as well, but instead of actively sensing and avoiding 
obstacles, each of the remaining vehicles may be configured 
to follow the one in front of it. This way there are not a large 
number of vehicles 102 in the convoy operating to indepen- 
dently find the best path to a destination. The vehicles 102 
may be configured so that if the lead vehicle 102 is disabled, 
the vehicle 102 behind the lead vehicle 102 may take over 
leading the convoy. 

FIGS. 10-12 show another vehicle 302 which includes the 
vehicle control system 100. The vehicle 302 may be a heavy 
duty tactical military vehicle. The vehicle 302 may be con- 
figured to function as a cargo truck (e.g., multiple configura- 
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tions of a cargo truck that can carry different types and 
amounts of cargo), a fuel servicing truck, a load handling 
system truck (e.g., includes a lift arm that loads pallets onto 
the vehicle), a bridge transport vehicle, and a recovery vehicle 
(e.g., wrecker vehicle) with cranes and winches. The vehicle 
302 may be configured so that it is capable of being trans- 
ported by a C130 airplane. 

The terms recited in the claims should be given their ordi- 
nary and customary meaning as determined by reference to 
relevant entries (e.g., definition of “plane” as a carpenter's 
tool would not be relevant to the use of the term “plane” when 
used to refer to an airplane, etc.) in dictionaries (e.g., consen- 
sus definitions from widely used general reference dictionar- 
ies and/or relevant technical dictionaries), commonly under- 
stood meanings by those in the art, etc, with the 
understanding that the broadest meaning imparted by any one 
or combination of these sources should be given to the claim 
terms (e.g., two or more relevant dictionary entries should be 
combined to provide the broadest meaning of the combina- 
tion of entries, etc.) subject only to the following exceptions: 
(а) ifaterm is used herein іп a manner more expansive than its 
ordinary and customary meaning, the term should be given its 
ordinary and customary meaning plus the additional expan- 
sive meaning, or (b) if a term has been explicitly defined to 
have a different meaning by reciting the term followed by the 
phrase “as used herein shall mean" or similar language (e.g., 
“herein this term means,” “as defined herein,” “for the pur- 
poses of this disclosure [the term] shall mean,” etc.). Refer- 
ences to specific examples, use of “i.e.” use of the word 
“invention,” etc., are not meant to invoke exception (b) or 
otherwise restrict the scope of the recited claim terms. 
Accordingly, the subject matter recited in the claims is not 
coextensive with and should not be interpreted to be coexten- 
sive with any particular embodiment, feature, or combination 
of features shown herein. This is true even if only a single 
embodiment of the particular feature or combination of fea- 
tures is illustrated and described herein. Thus, the appended 
claims should be read to be given their broadest interpretation 
in view ofthe prior art and the ordinary meaning of the claim 
terms. 

As used herein, spatial or directional terms, such as “left,” 
"right," “front,” “back,” and the like, relate to the subject 
matter as it is shown in the drawing FIGS. However, it is to be 
understood that the subject matter described herein may 
assume various alternative orientations and, accordingly, 
such terms are not to be considered as limiting. Furthermore, 
as used herein (i.e., in the claims and the specification), 
articles such as “the,” “a,” and “an” can connote the singular 
or plural. Also, as used herein, the word “ог” when used 
without a preceding "either" (or other similar language indi- 
cating that “or” is unequivocally meant to be exclusive—e.g., 
only one of x or y, etc.) shall be interpreted to be inclusive 
(e.g., “x or y" means one or both x or y). Likewise, as used 
herein, the term “and/or” shall also be interpreted to be inclu- 
sive (e.g., “x and/or y" means one or both x or y). In situations 
where “and/or” or “ог” are used as a conjunction for a group 
of three or more items, the group should be interpreted to 
include one item alone, all of the items together, or any 
combination or number of the items. Moreover, terms used in 
the specification and claims such as have, having, include, 
and including should be construed to be synonymous with the 
terms comprise and comprising. 

Unless otherwise indicated, all numbers or expressions, 
such as those expressing dimensions, physical characteris- 
tics, etc. used in the specification are understood as modified 
in all instances by the term “about.” At the very least, and not 
asanattempt to limit the application ofthe doctrine of equiva- 
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lents to the claims, each numerical parameter recited in the 
specification or claims which is modified by the term “about” 
should at least be construed in light of the number of recited 
significant digits and by applying ordinary rounding tech- 
niques. Moreover, all ranges disclosed herein are to be under- 
stood to encompass any and all subranges subsumed therein. 
For example, a stated range of 1 to 10 should be considered to 
include any and all subranges between and inclusive of the 
minimum value of 1 and the maximum value of 10; that is, all 
subranges beginning with a minimum value of 1 or more and 
ending with a maximum value of 10 or less (e.g., 5.5 to 10). 


What is claimed is: 

1. A vision system comprising: 

a processor; and 

a plurality of cameras each of which is configured to be 
communicatively coupled to the processor, the plurality 
of cameras including at least three cameras; 

wherein the processor is configured to select input from 
different combinations of the plurality of cameras based 
on a parameter that is associated with at least one of a 
status of a vehicle and a surrounding environment of the 
vehicle, such that the processor is configured to select a 
first combination of two or more of the plurality of 
cameras when the parameter is within a first value range 
and a second combination of two or more of the plurality 
of cameras when the parameter is within a second value 
range, 
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wherein the first combination of cameras includes two 
cameras spaced apart by a first distance and the second 
combination of cameras includes two cameras spaced 
apart by a second distance, the first distance being larger 
than the second distance, 
wherein the vision system is used to identify one or more 
objects at varying distances from the plurality of cam- 
eras, and 
wherein the processor receives input from the first combi- 
nation of cameras when the one or more objects are at a 
third distance and the processor receives input from the 
second combination of cameras when the one or more 
objects are at a fourth distance, the third distance being 
larger than the fourth distance. 
2. The vision system of claim 1, wherein the parameter 
includes a speed of the vehicle. 
3. The vision system of claim 1, wherein the parameter 
includes a steering angle of the vehicle. 
4. The vision system of claim 1, wherein the cameras are 
each a different type of camera. 
5. The vision system of claim 1, wherein at least one ofthe 
cameras includes an infrared camera. 
6. The vision system of claim 1, wherein the vision system 
is configured for use with an autonomous vehicle. 
7. The vision system of claim 1, wherein at least one of the 
plurality of cameras is a video camera. 
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